Systems and methods for providing features in a friction display including a haptic effect based on a color and a degree of shading

ABSTRACT

A touch-enabled device can simulate one or more features in a touch area. Features may include, but are not limited to, changes in texture and/or simulation of boundaries, obstacles, or other discontinuities in the touch surface that can be perceived through use of an object in contact with the surface. Systems include a sensor configured to detect a touch in a touch area when an object contacts a touch surface, an actuator, and one or more processors. The processor can determine a position of the touch using the sensor and select a haptic effect to generate based at least in part on the position, the haptic effect selected to simulate the presence of a feature at or near the determined position. Some features are simulated by varying the coefficient of friction of the touch surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims priority to and is a continuation of U.S.patent application Ser. No. 12/696,893, entitled “Systems and Methodsfor Providing Features in a Friction Display Wherein a Haptic Effect isConfigured to Vary the Coefficient of Friction,” filed Jan. 29, 2010,which claims priority to U.S. Provisional Patent Application No.61/159,482, entitled “Locating Features Using a Friction Display,” filedMar. 12, 2009, both of which are hereby incorporated by reference hereinin their entirety.

U.S. patent application Ser. No. 12/696,893 claims priority to U.S.Provisional Patent Application No. 61/262,041, entitled “System andMethod for Increasing Haptic Bandwidth in an Electronic Device” filedNov. 17, 2009, which is incorporated by reference herein in itsentirety.

U.S. patent application Ser. No. 12/696,893 claims priority to U.S.Provisional Patent Application No. 61/262,038, entitled “Friction RotaryDevice for Haptic Feedback” filed Nov. 17, 2009, which is incorporatedby reference herein in its entirety.

This patent application is related to U.S. patent application Ser. No.12/697,010 filed the same day as U.S. patent application Ser. No.12/696,893 and entitled “Systems and Methods for a Texture Engine,”which is incorporated by reference herein in its entirety.

This patent application is related to U.S. patent application Ser. No.12/697,042, filed the same day as U.S. patent application Ser. No.12/696,893 and entitled “Systems and Methods for Using MultipleActuators to Realize Textures,” which is incorporated by referenceherein in its entirety.

This patent application is related to U.S. patent application Ser. No.12/697,037, filed the same day as U.S. patent application Ser. No.12/696,893 and entitled “Systems and Methods for Using Textures inGraphical User Interface Widgets,” which is incorporated by referenceherein in its entirety.

This patent application is related to U.S. patent application Ser. No.12/696,900, filed the same day as U.S. patent application Ser. No.12/696,893 and entitled “Systems and Methods for Friction Displays andAdditional Haptic Effects,” which is incorporated by reference herein inits entirety.

This patent application is related to U.S. patent application Ser. No.12/696,908, filed the same day as U.S. patent application Ser. No.12/696,893 and entitled “Systems and Methods for Interfaces FeaturingSurface-Based Haptic Effects,” which is incorporated by reference hereinin its entirety.

BACKGROUND

Touch-enabled devices have been increasingly popular. For instance,mobile and other devices may be configured with touch-sensitive displaysso that a user can provide input by touching portions of thetouch-sensitive display. As another example, a touch-enabled surfaceseparate from a display may be used for input, such as a trackpad,mouse, or other device.

For example, a user may touch a portion of the display or surface thatis mapped to an on-screen graphical user interface, such as a button orcontrol. As another example, a gesture may be provided, such as asequence of one or more touches, drags across the surface, or otherrecognizable patterns sensed by the device. Although touch-enableddisplays and other touch-based interfaces have greatly enhanced devicefunctionality, drawbacks remain. For instance, even if a keyboard isdisplayed on a screen, a user accustomed to a physical keyboard may nothave the same experience while using the touch-enabled device.

SUMMARY

Embodiments of the present invention include devices featuringsurface-based haptic effects that simulate one or more features in atouch area. Features may include, but are not limited to, changes intexture and/or simulation of boundaries, obstacles, or otherdiscontinuities in the touch surface that can be perceived through useof an object in contact with the surface. Devices includingsurface-based haptic effects may be more user friendly and may provide amore compelling user experience.

In one embodiment, a system includes a sensor configured to detect atouch in a touch area when an object contacts a touch surface, anactuator, and one or more processors. The touch area may correspond to adisplay area and/or another surface with which a user interacts via anobject such as a finger or pen. The processor can be configured todetermine a position of the touch based on data from the sensor andselect a haptic effect to generate based at least in part on theposition, the haptic effect selected to simulate the presence of afeature at or near the determined position. Further, the processor cantransmit a haptic signal to generate the identified haptic effect usingthe actuator. The actuator can be coupled to the touch surface and canbe configured to receive a haptic signal generated by the processor andoutput a haptic effect in response. In some embodiments, selecting thehaptic effect comprises determining a variation in the coefficient offriction that will simulate the presence of the feature at or near thedetermined position.

These illustrative embodiments are mentioned not to limit or define thelimits of the present subject matter, but to provide examples to aidunderstanding thereof. Illustrative embodiments are discussed in theDetailed Description, and further description is provided there.Advantages offered by various embodiments may be further understood byexamining this specification and/or by practicing one or moreembodiments of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure is set forth more particularly in theremainder of the specification. The specification makes reference to thefollowing appended figures.

FIG. 1A shows an illustrative system for providing surface-based hapticeffects.

FIG. 1B shows an external view of one embodiment of the system shown inFIG. 1A.

FIG. 1C illustrates an external view of another embodiment of the systemshown in FIG. 1A.

FIGS. 2A-2B illustrate an example of simulating a feature using asurface-based haptic effect.

FIGS. 3A-3B depict an illustrative hardware architecture for varying thecoefficient of friction of a surface.

FIGS. 4A-4B depict another illustrative hardware architecture forvarying the coefficient of friction of a surface.

FIG. 5 is a flowchart showing an exemplary method for providing asimulated feature by using surface-based haptic effects.

FIGS. 6A-6D each depict an illustrative simulated feature.

FIG. 7 is a flowchart showing an exemplary method for providing asimulated feature by using a reference file.

FIG. 8 is an example of a reference file including data for simulatingfeatures of a touch surface.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternativeillustrative embodiments and to the accompanying drawings. Each exampleis provided by way of explanation, and not as a limitation. It will beapparent to those skilled in the art that modifications and variationscan be made. For instance, features illustrated or described as part ofone embodiment may be used on another embodiment to yield a stillfurther embodiment. Thus, it is intended that this disclosure includemodifications and variations as come within the scope of the appendedclaims and their equivalents.

Illustrative Example of a Device Using a Variable Friction Interface

One illustrative embodiment of the present invention comprises acomputing system such as an iPod® portable music device or iPhone®mobile device, both available from Apple Inc. of Cupertino, Calif., or aZune® portable device, available from Microsoft Corporation of Redmond,Wash. The computing system can include and/or may be in communicationwith one or more sensors, such as an accelerometer, as well as sensors(e.g., optical, resistive, or capacitive) for determining a location ofa touch relative to a display area corresponding in this example to thescreen of the device.

As the user interacts with the device, one or more actuators are used toprovide tactile effects. For example, as a user moves a finger acrossthe device, the coefficient of friction of the screen can be variedbased on the position, velocity, and/or acceleration of the finger.Depending on how the friction is varied, the user may perceive a featurein the touch surface that would not otherwise be perceived in the samemanner (or at all) if the surface friction were not varied. As aparticular example, the friction may be varied so that the userperceives a bump, border, or other obstacle corresponding to an edge ofan on-screen button. As will be discussed in further detail below,varying the coefficient of friction can be used in any number of ways toprovide information to a user. Additionally, the presence of a featurein the touch surface can be simulated using effects in addition to orinstead of varying the coefficient of friction.

Illustrative Systems for Simulating Features by Providing Surface-BasedHaptic Effects

FIG. 1A shows an illustrative system 100 for providing a surface-basedhaptic effect. Particularly, in this example, system 100 comprises acomputing device 101 featuring a processor 102 interfaced with otherhardware via bus 106. A memory 104, which can comprise any suitabletangible (and non-transitory) computer-readable medium such as RAM, ROM,EEPROM, or the like, embodies program components that configureoperation of the computing device. In this example, computing device 101further includes one or more network interface devices 110, input/output(I/O) interface components 112, and additional storage 114.

Network device(s) 110 can represent any components that facilitate anetwork connection. Examples include, but are not limited to, wiredinterfaces such as Ethernet, USB, IEEE 1394, and/or wireless interfacessuch as IEEE 802.11, Bluetooth, or radio interfaces for accessingcellular telephone networks (e.g., transceiver/antenna for accessing aCDMA, GSM, UMTS, or other mobile communications network).

I/O components 112 may be used to facilitate connection to devices suchas a one or more displays, keyboards, mice, speakers, microphones,and/or other hardware used to input data or output data. Storage 114represents nonvolatile storage such as magnetic, optical, or otherstorage media included in device 101.

System 100 further includes a touch surface 116, which is in thisexample integrated into device 101. Touch surface 116 represents anysurface that is configured to sense tactile input of a user. One or moresensors 108 are configured to detect a touch in a touch area when anobject contacts a touch surface and provide appropriate data for use byprocessor 102. Any suitable number, type, or arrangement of sensors canbe used. For example, resistive and/or capacitive sensors may beembedded in touch surface 116 and used to determine the location of atouch and other information, such as pressure. As another example,optical sensors with a view of the touch surface may be used todetermine the touch position.

In this example, an actuator 118 in communication with processor 102 iscoupled to touch surface 116. In some embodiments, actuator 118 isconfigured to output a haptic effect varying a coefficient of frictionof the touch surface in response to a haptic signal. Additionally oralternatively, actuator 118 may provide vibrotactile haptic effects thatmove the touch surface in a controlled manner. Some haptic effects mayutilize an actuator coupled to a housing of the device, and some hapticeffects may use multiple actuators in sequence and/or in concert. Forexample, the coefficient of friction can be varied by vibrating thesurface at different frequencies. Different combinations/sequences ofvariance can be used to simulate the feeling of a texture.

Although a single actuator 118 is shown here, embodiments may usemultiple actuators of the same or different type to vary the coefficientof friction of the touch surface. For example, a piezoelectric actuatoris used in some embodiments to displace some or all of touch surface 116vertically and/or horizontally at ultrasonic frequencies, such as byusing an actuator moving at frequencies greater than 20 kHz in someembodiments. In some embodiments, multiple actuators such as eccentricrotating mass motors and linear resonant actuators can be used alone orin concert to provide different textures and other haptic effects.

Turning to memory 104, exemplary program components 124, 126, and 128are depicted to illustrate how a device can be configured in someembodiments to provide a variable-friction display. In this example, adetection module 124 configures processor 102 to monitor touch surface116 via sensor(s) 108 to determine a position of a touch. For example,module 124 may sample sensor 108 in order to track the presence orabsence of a touch and, if a touch is present, to track the location,path, velocity, acceleration, pressure and/or other characteristics ofthe touch over time.

Haptic effect determination module 126 represents a program componentthat analyzes data regarding touch characteristics to select a hapticeffect to generate. Particularly, module 126 comprises code thatdetermines, based on the location of the touch, a simulated feature ofthe touch surface to generate and code that selects one or more hapticeffects to provide in order to simulate the feature. For example, someor all of the area of touch surface 116 may be mapped to a graphicaluser interface. Different haptic effects may be selected based on thelocation of a touch in order to simulate the presence of the feature byvarying the friction of touch surface 116 so that the feature is feltwhen a corresponding representation of the feature is seen in theinterface. However, haptic effects may be provided via touch surface 116even if a corresponding element is not displayed in the interface (e.g.,a haptic effect may be provided if a boundary in the interface iscrossed, even if the boundary is not displayed).

Haptic effect generation module 128 represents programming that causesprocessor 102 to generate and transmit a haptic signal to actuator(s)118 to generate the selected haptic effect at least when a touch isoccurring. For example, generation module 128 may access storedwaveforms or commands to send to actuator 118. As another example,haptic effect generation module 128 may receive a desired coefficient offriction and utilize signal processing algorithms to generate anappropriate signal to send to actuator(s) 118. As a further example, adesired texture may be indicated along with target coordinates for thetexture and an appropriate waveform sent to one or more actuators togenerate appropriate displacement of the surface (and/or other devicecomponents) to provide the texture. The feature may be simulated atleast by varying a coefficient of friction of touch surface 116. Someembodiments may utilize multiple actuators in concert to simulate afeature. For instance, a variation in friction may be used to simulatecrossing a boundary between simulated piano keys while a vibrotactileeffect simulates the response of each key as it is pressed.

A touch surface may or may not overlay (or otherwise correspond to) adisplay, depending on the particular configuration of a computingsystem. In FIG. 1B, an external view of a computing system 100B isshown. Computing device 101 includes a touch-enabled display 116 thatcombines a touch surface and a display of the device. The touch surfacemay correspond to the display exterior or one or more layers of materialabove the actual display components.

FIG. 1C illustrates another example of a touch-enabled computing system100C in which the touch surface does not overlay a display. In thisexample, a computing device 101 features a touch surface 116 which maybe mapped to a graphical user interface provided in a display 122 thatis included in computing system 120 interfaced to device 101. Forexample, computing device 101 may comprise a mouse, trackpad, or otherdevice, while system 120 may comprise a desktop or laptop computer,set-top box (e.g., DVD player, DVR, cable television box), or anothercomputing system. As another example, touch surface 116 and display 122may be included in the same device, such as a touch-enabled trackpad ina laptop computer featuring display 122. Whether integrated with adisplay or otherwise, the depiction of planar touch surfaces in theexamples herein is not meant to be limiting. Other embodiments includecurved or irregular touch-enabled surfaces that are further configuredto provide surface-based haptic effects.

FIGS. 2A-2B illustrate an example of simulating a feature using asurface-based haptic effect. FIG. 2A is a diagram illustrating anexternal view of a system 200 comprising a computing device 201 thatfeatures a touch-enabled display 202. FIG. 2B shows a cross-sectionalview of device 201. Device 201 may be configured similarly to device 101of FIG. 1A, though components such as the processor, memory, sensors,and the like are not shown in this view for purposes of clarity.

As can be seen in FIG. 2B, device 201 features a plurality of actuators218 and an additional actuator 222. Actuator 218-1 may comprise anactuator configured to impart vertical force to display 202, while 218-2may move display 202 laterally. In this example, the actuators arecoupled directly to the display, but it should be understood that theactuators could be coupled to another touch surface, such as a layer ofmaterial on top of display 202. Additional actuator 222 may be coupledto a housing containing the components of device 201. In the examples ofFIGS. 2A-2B, the area of display 202 corresponds to the touch area,though the principles could be applied to a touch surface completelyseparate from the display.

In one embodiment, actuators 218 each comprise a piezoelectric actuator,while additional actuator 222 comprises an eccentric rotating massmotor, a linear resonant actuator, or another piezoelectric actuator.Actuator 222 can be configured to provide a vibrotactile haptic effectin response to a haptic signal from the processor. The vibrotactilehaptic effect can be utilized in conjunction with surface-based hapticeffects and/or for other purposes.

In some embodiments, either or both actuators 218-1 and 218-2 cancomprise an actuator other than a piezoelectric actuator. Any of theactuators can comprise a piezoelectric actuator, an electromagneticactuator, an electroactive polymer, a shape memory alloy, a flexiblecomposite piezo actuator (e.g. an actuator comprising a flexiblematerial), electrostatic, and/or magnetostrictive actuators, forexample. Additionally, a single actuator 222 is shown, although multipleother actuators can be coupled to the housing of device 201 and/or otheractuators 222 may be coupled elsewhere. Device 201 may feature multipleactuators 218-1/218-2 coupled to the touch surface at differentlocations, as well.

Turning back to FIG. 2A, as shown at 220, a finger moves to encounter asimulated feature 230. In this example, a haptic effect is selected foroutput based on the position of a touch represented by movement offinger 226 downward to position 228. Particularly, as can be seen inFIG. 4B, actuators 218-1, 218-2, and/or 222 are provided withappropriate haptic signals to provide surface-based haptic feedback asindicated at 232, 234, and 236. The different cross-hatching is intendedto represent different “feel” of the touch surface due to the actuators.For instance, 232, 234, and 236 can represent variations in the textureor coefficient of friction of the touch surface that generate thedesired haptic effect. In one embodiment, the feeling of a box can besimulated by having a first area 232 of higher friction followed by asecond area 234 of lower friction and a third area 236 of higherfriction.

FIGS. 3A-3B depict an illustrative hardware architecture for varying thecoefficient of friction of a surface. In this example, the touch surfacecomprises glass panel 302, although another transparent material (ornon-transparent) material could be used. For instance, rather thanglass, a touchpad (i.e. touch-sensitive device) could be used. A pair ofpiezoelectric benders 318-1 and 318-2 are bonded to the glass. Use ofglass or another transparent material along with free space between thepiezoelectric benders can allow for use of a display (not shown) beneaththe glass. In some embodiments, the piezoelectric benders can becommanded to reduce the static coefficient of friction of glass 302 by42%. Some embodiments utilize a bipolar pulse width modulated signal at24 kHz, with varying amplitude used to vary the coefficient of friction.As an example, voltage can vary between −80 and +80 Volts at a frequencyabove 20 kHz, with friction variation from 0 to 60% depending on voltagemagnitude (or PWM magnitude to produce the voltage magnitude). Theseexample voltage, frequency, and variation ranges are for purposes ofexample only and are not intended to be limiting.

FIGS. 4A-4B depict another illustrative hardware architecture 400 forvarying the coefficient of friction of a surface. In this example, apiezo buzzer presents a piezo surface 402 as part of computing device401. For instance, one embodiment includes a 0.6 mm thick buzzer with a25 mm diameter. As can be seen in FIG. 4B, the buzzer comprises a layerof piezoceramic material 402A and a metal disk 402B; in this embodiment,both are 0.3 mm thick. The static coefficient of friction may be reducedup to 88% from the original friction value when the buzzer is notactivated. In this example, the surface is shown as round, although theprinciple could be applied to other shapes/surfaces.

Illustrative Methods for Simulating Features by Providing Surface-BasedHaptic Effects

FIG. 5 is a flowchart showing an illustrative method 500 for providingan interface with surface-based haptic effects. Block 502 representsdetermining a position of a touch in a touch area. For example, aprocessor may utilize one or more sensors embedded in or viewing atouch-enabled display or surface to track a position of a touch on thesurface. Based on the current and/or past position of the touch, apresent position or predicted position can be determined. As an example,a touch position may be provided in pixel coordinates or anothercoordinate system for the touch area. If velocity/acceleration isdetermined, the touch position may be associated with a vector or otherinformation pertaining to position and motion of the touch.

Block 504 represents determining one or more desired features tosimulate. In some embodiments, the computing system may simply determinewhether or not the touch occurs at a location at which the actuator(s)are to be driven, with the desired feature and haptic signals determinedin real time. However, in additional embodiments, the current pixellocation and/or a projected pixel location for the touch based on avelocity of the touch can be compared to a bitmap specifying desiredhaptic effects for various pixel positions. Based on the desired hapticeffect(s), suitable haptic signals can be accessed/generated to providethe output specified in the bitmap.

As another example, a current or projected location of a touch can becompared to data identifying the location of graphical user interface(GUI) features such as controls, textual content, boundaries, and thelike. Then, if a GUI feature is identified at the location, dataassociating one or more haptic effects to the feature can be accessed.For instance, a processor may track the location of a touch anddetermine the touch is at or approaching a position in the touch areamapped to a particular control (e.g., a button) in the graphical userinterface. The processor can then consult a listing of interfaceelements to determine a haptic effect (e.g., a texture, a frictionvariation) associated with the button and, based on the haptic effect,take further actions to generate the haptic effect.

As a further example, the feature may comprise a texture associated withthe current or projected location. For instance, a portion of the toucharea may be identified as having a particular texture, such as “fur.”When the touch is determined to be at the portion, the computing devicecan determine that the “fur” feature is desired.

Block 506 represents accessing or generating one or more haptic signalsto generate the selected haptic effect(s). For example, a processor mayaccess drive signals stored in memory and associated with particularhaptic effects. As another example, a signal may be generated byaccessing a stored algorithm and inputting parameters associated with aneffect. For example, an algorithm may output data for use in generatinga drive signal based on amplitude and frequency parameters. As anotherexample, a haptic signal may comprise data sent to an actuator to bedecoded by the actuator. For instance, the actuator may itself respondto commands specifying parameters such as amplitude and frequency.

Block 508 represents transmitting the haptic signal to the actuator(s)to generate the desired effect(s). For instance, if an analog drivesignal is to be provided, a processor can utilize an onboard D/Aconverter to create the signal. If a digital command is provided to theactuator, an appropriate message can be generated by an I/O bus of theprocessor, with the actuator itself including sufficient processingcapability to provide the desired output. The haptic effect may be feltat the point of the touch and/or elsewhere.

In some embodiments, a baseline haptic signal may be sent to theactuator(s) to generate an ambient haptic effect even in the absence ofa selected haptic effect in order to enhance the range of potentialeffects the device can produce. Thus, transmitting a haptic signal maycomprise sending a stop, command, a zero or minimal signal to theactuator, or another suitable signal to the actuator to reduce intensityin order to the effect of the actuator and thus increase the friction,such as increasing to a level near or at the coefficient of friction forthe touch surface when static.

As an example, use of certain actuators, such as piezoelectricactuators, may allow for reduction in the coefficient of friction of atouch surface but not an increase in the coefficient of friction. Toprovide a range of options, a baseline signal may be provided so thatthe “ordinary” friction level of the touch surface is below thecoefficient of friction the touch surface would have when static.Accordingly, haptic effects may be defined with respect to the baseline,rather than static, value. If maximum friction is desired, a “zero”signal may be sent to the piezoelectric actuator to stop movement of thesurface.

FIGS. 6A-6D each depict an illustrative simulated feature. FIG. 6A showsa simplified example in which the white area represents an area wherepiezoelectric or other actuators will be activated, such as by using anon-zero voltage PWM signal. For example, the white area may correspondto a virtual button in the middle of a touch pad, where a user's finger(or another object in contact with a surface) will encounter a lowerfriction value. FIG. 6B represents an inverses situation—thefinger/object may navigate freely in the white area, but may be slowedor stopped at the high friction (black) area. This may, for example,allow a user to more easily locate a button or other location in thetouch area.

FIG. 6C illustrates a simulated feature comprising a plurality ofgrooves. As a user's finger or another object moves horizontally acrossthe stripes, the finger/object will encounter increasing and decreasingfriction that is perceived as a series of grooves.

As was noted above, a computing system comprising a touch surfaceconfigured to provide surface-based haptic effects may determine effectsand signals in real time. For example, for any of FIGS. 6A-6D, thesystem may first determine if the touch position is inside the circleand, if so, provide a suitable output value (FIG. 6A) or cease output(FIG. 6B). Similarly, the system may provide the feature of FIG. 6C bydetermining if the touch occurs in an area with desired high-frictionand, if so, drive the actuator(s).

FIG. 6D presents a more complex pattern. For instance, the pattern inFIG. 6D may correspond to desired features associated with an array ofkeys, such as an array of mobile phone keys, a simulated keyboard, orother controls. Although real time rendering could be used for any ofFIGS. 6A-6D, more complex logic may be needed to render each specificcircle/button in the pattern. These and even more arbitrary patters mayincrease the complexity of programming and computation time. Thus, insome embodiments, the surface-based haptic effects can be determinedahead of time and stored in a file. At runtime, the file can be accessedbased on a touch position to allow for faster determination andgeneration of appropriate haptic signals. For FIG. 6D, such a file couldinclude data to drive the actuators to provide a first haptic effect(e.g., high friction) when the touch position is mapped to the circles,and the file could include data to drive the actuators to provide asecond effect (e.g., low friction) when the touch position is mapped toa location outside the circles.

FIG. 7 is a flowchart showing an exemplary method 700 for providing asimulated feature by creating and using a reference file. FIG. 8 showsan example of a reference file comprising an array of pixels. Blocks 702and 704 represent preprocessing—activities that occur prior to use of areference file to determine a haptic effect. In this example, a singlereference file is used to determine friction values. In practice, areference file may provide other data for use in generating hapticeffects in addition to or instead of generating variances in friction.Additionally, a “reference file” may comprise multiple files usedtogether.

Block 702 represents creating a layout of a location and block 704represents storing the layout in an image file, such as an array ofpixels in a bitmap or other image file. For example, arbitrary shapesmay be “drawn” in order to specify desired friction values. In FIG. 8,white pixels are shown to indicate where no friction adjustment isintended, while shaded pixels indicate a value of a desired coefficientof friction or even a value usable to drive an actuator (e.g., a desiredPWM voltage level, frequency, etc.). Alternatively, white pixels mayindicate maximum drive, while various degrees of shading indicate lowerdrive values, with black representing zero drive. In an embodiment,white pixels and black pixels only are used, with the colorscorresponding to on/off states of the actuators of a device.

In this example, different degrees of shading are represented bycross-hatching. In practice, each pixel may comprise multiple values(e.g., each pixel may have an RGB value), with the multiple valuesproviding different data, such as drive levels for different actuatorsand the like. Additionally, a reference file may include multiple layersfor specifying various parameters for each pixel position. This exampleshows a relatively small number of pixels; in practice, the array maycomprise thousands or millions of pixels.

Shape 802 comprises a solid circle. Shape 804 also comprises a circle,but is provided to indicate that the image file can be used to specifymultiple levels of friction (or other haptic effects). For example,transition areas between low and high (or high and low) friction can beprovided through use of different shadings, such as the transition fromlow shading 806 to moderate shading 808 and finally to full shading 810.The transition may correspond to an increasing friction level (ordecreasing friction level) as the center of circle 804 is approached.

In some embodiments, transitions can be specified when the layout fileis created. In some instances, the transitions may be used to provideoffsets of friction values with respect to a visual layout. For example,returning briefly to FIG. 6B, a solid shape such as the circle of FIG.6B may be provided in a graphical user interface. The correspondingfriction image may more closely resemble a scaled version of circle 804of FIG. 8, providing “fuzzy” edges to represent a transition effect. Onemethod of generating such an image could comprise using an image of aninterface and applying a blur or other filter, with the pixel levelsadjusted after blurring to provide a desired response when used togenerate/select haptic signals.

Returning to FIG. 7 and method 700, once a reference file is created, itcan be loaded into memory and read as shown at block 706 to determinefriction values. For example, some or all of the pixel array may bemaintained in working memory of a processor carrying out a positiondetection and feature simulation routine. In an embodiment, the pixelarray is distributed alongside a corresponding image of a graphical userinterface. In additional embodiments, the pixel array is a layer orcomponent of the graphical user interface image, and in furtherembodiments the array is a separate file not associated with a graphicaluser interface.

Block 708 represents determining a position of a touch. For example, asensor may provide data used to determine a pixel position of a touch inan array of pixels mapped to a touch area. Non-pixel coordinates may beused in identifying the location of a touch, with appropriate transformsused during the mapping step below.

Block 710 represents mapping the touch position to an entry (or entries)in the image file. For instance, the touch area may be mapped directlyso that a touch at pixel (x,y)=(10, 12) results in accessing one or morepixel values in the image at image (x,y)=(10,12). However, more complexmappings may be used. For example, a touch position and velocity may beused to map a pixel value in the touch area to a different pixel valuein the image file. For instance, the size of the touch area and the sizeof the pixel array may differ, with a scaling factor used to map touchlocations to pixel values.

Block 712 represents activating one or more actuators to provide asurface-based haptic effect based at least in part on data from theimage file. For instance, the pixel value in the image file may bemapped to a desired coefficient of friction. A device carrying outmethod 700 may determine, based on the pixel position and the desiredcoefficient of friction, a suitable signal or signals to send to one ormore actuators to generate the desired coefficient of friction. Asanother example, the pixel value may indicate a drive signal moredirectly, such as a voltage/amplitude/frequency value or offset for aPWM signal to be sent to a piezoelectric actuator. Data of the array mayalso be configured for use in generating a drive signal for another typeof actuator.

As a more complex example, each pixel address may be associated withthree intensity values (i.e., RGB). Each of the three intensity valuescan be associated with a signal intensity/frequency for a correspondingactuator in some embodiments. As another example, some values mayspecify intensity and others specify duration of operation for the sameactuator. As a further example, different pixel intensity values may becorrelated to different desired textures or components used to driveactuators to simulate a single texture.

Method 700 may determine touch locations mapped to multiple pixels inthe image file. For example, a large touch may correspond to a range ofpixel addresses in the image file. Friction or other values from therange of pixel addresses may be considered together, or analysis may bemade to “pinpoint” the touch location and use a value from acorresponding single pixel address.

In some embodiments, a computing device featuring a touch surface withsurface-based haptic effects can output different surface-based hapticeffects based on sequences of inputs. Thus, the simulated features ofthe touch surface can vary based on a state of a device associated withthe surface. In some embodiments, this can be implemented using areference file with multiple layers; each layer can correspond to aparticular state. The states can be changed based on various inputconditions, for instance.

For example, a touch surface may be configured to act as a keypad, suchas on a mobile device. The keypad may feature three rows of keyscorresponding to numbers 1-9 and a fourth row with “0,” “*” and “#”keys. For an initial state, the touch surface may be configured toprovide a centering feature, such as a higher friction level at the “5”key than in the remainder of the layout.

The computing device can be configured to change the state of the touchsurface in response to user input based on tracking the input relativeto the touch-sensitive area. For example, once the system determinesthat the user has found the “5” key, e.g. by detecting touching,hovering, or other activity indicating that the key has been located(but not necessarily selected), the surface-based effects can beprovided based on a different state. If a multi-layer reference file isused, for example, a different layer can be loaded into memory. In thesecond state, for instance, boundaries between keys can be provided sothat a user can proceed from the center to a desired key without theneed for visual feedback (although, of course, visual, auditory, orother feedback can be provided alongside any embodiments of the presentsubject matter).

Other Illustrative Embodiments of Simulated Features

Surface-based haptic effects may take any suitable form, including, butnot limited to, effects based on varying the coefficient of friction ofthe touch surface. As another example, vibrotactile effects may be used,such as vibrations or series of vibrations. Vibrotactile effects and/orvariations in friction may be used to simulate the feeling of distinctfeatures, such as boundaries or obstacles. For example, a boundary oredge may be simulated by an increase in friction, with the frictiondecreasing if the boundary is crossed (in some instances) as notedabove.

Features simulated using the touch surface can comprise anydiscontinuity, including, but not limited to, simulated gaps,protrusions, obstacles, and the like. Additionally or alternatively, asimulated feature can comprise an area with a changed coefficient offriction. For example, some haptic effects may comprise variations inthe friction of the touch surface—some portions may be rendered“slicker” or “rougher” than others. In some embodiments, a simulatedfeature includes a texture simulated by varying the coefficient offriction of the surface in a controlled manner.

Additional detail regarding generation and use of textures can be foundin U.S. patent application Ser. Nos. 12/697,010, 12/697,042, and12/697,037, referenced above and entitled “Systems and Methods for aTexture Engine,” “Systems and Methods for Using Multiple Actuators toRealize Textures,” and “Systems and Methods for Using Textures inGraphical User Interface Widgets,” respectively. For instance, patternsof differing friction or patterns of vibration may be provided to mimicthe feeling of textures such as brick, rocks, sand, grass, fur, variousfabric types, water, molasses, and other fluids, leather, wood, ice,lizard skin, metals, and other texture patterns. Other textures notanalogous to real-world textures may also be used, such ashigh-magnitude vibrotactile or other feedback when a “danger” texture isdesired.

The informational content or meaning of surface-based haptic effects canvary in various embodiments. For example, effects may be used toidentify particular portions of a touch surface mapped to areas in agraphical user interface, simulated keys or other controls, or may beprovided for aesthetic or entertainment purposes (e.g., as part of adesign and/or in a game). Effects may be provided for communicationpurposes as well. For example, Braille or other tactile-basedcommunications methods can be facilitated.

General Considerations

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

Embodiments in accordance with aspects of the present subject matter canbe implemented in digital electronic circuitry, in computer hardware,firmware, software, or in combinations of the preceding. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises or has access to a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs including a sensor samplingroutine, a haptic effect selection routine, and suitable programming toproduce signals to generate the selected haptic effects as noted above.

Such processors may comprise a microprocessor, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC),field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example tangible computer-readable media, that may storeinstructions that, when executed by the processor, can cause theprocessor to perform the steps described herein as carried out, orassisted, by a processor. Embodiments of computer-readable media maycomprise, but are not limited to, all electronic, optical, magnetic, orother storage devices capable of providing a processor, such as theprocessor in a web server, with computer-readable instructions. Otherexamples of media comprise, but are not limited to, a floppy disk,CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configuredprocessor, all optical media, all magnetic tape or other magnetic media,or any other medium from which a computer processor can read. Also,various other devices may include computer-readable media, such as arouter, private or public network, or other transmission device. Theprocessor, and the processing, described may be in one or morestructures, and may be dispersed through one or more structures. Theprocessor may comprise code for carrying out one or more of the methods(or parts of methods) described herein.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

What is claimed:
 1. A system comprising: a processor configured to:determine a position of a user interaction on a touch surface; map theposition of the user interaction to a location on a graphical userinterface associated with the touch surface; determine a haptic effectbased in part on a color and a degree of shading at the location on thegraphical user interface, wherein the haptic effect is configured tovary a coefficient of friction on the touch surface at the position ofthe user interaction, and the degree of shading is associated with thevariance of the coefficient of friction; and transmit a haptic signal toa haptic output device configured to output the haptic effect.
 2. Thesystem of claim 1, wherein the haptic effect comprises a texture at ornear the position of the user interaction.
 3. The system of claim 1,wherein the haptic effect comprises a boundary between a first regionand a second region in the graphical user interface.
 4. The system ofclaim 3, wherein the boundary comprises a boundary between two simulatedkeys of a keyboard.
 5. The system of claim 1, wherein the processor isfurther configured to determine a velocity of the user interaction anddetermine the haptic effect based at least in part on the velocity. 6.The system of claim 1, wherein the processor is further configured todetermine a pressure of the user interaction and determine the hapticeffect based at least in part on the pressure.
 7. The system of claim 1,wherein the haptic output device is configured to displace the touchsurface vertically, laterally, or vertically and laterally at anultrasonic frequency.
 8. The system of claim 7, wherein the hapticoutput device comprises one or more of: a piezoelectric actuator, anelectromagnetic actuator, an electroactive polymer, a shape memoryalloy, or a composite.
 9. The system of claim 1, wherein varying thecoefficient of friction of the touch surface comprises increasing thecoefficient of friction of the touch surface.
 10. A method, comprising:determining a position of a user interaction on a touch surface; mappingthe position of the user interaction to a location on a graphical userinterface associated with the touch surface; determining a haptic effectbased in part on a color and a degree of shading at the location on thegraphical user interface, wherein the haptic effect is configured tovary a coefficient of friction on the touch surface at the position ofthe user interaction, and the degree of shading is associated with thevariance of the coefficient of friction; and transmitting a hapticsignal to a haptic output device configured to output the haptic effect.11. The method of claim 10, wherein the haptic effect comprises atexture at or near the position of the user interaction.
 12. The methodof claim 10, wherein the haptic effect comprises a boundary between afirst region and a second region in the graphical user interface. 13.The method of claim 12, wherein the boundary comprises a boundarybetween two simulated keys of a keyboard.
 14. The method of claim 10,further comprising determining a velocity of the user interaction anddetermining the haptic effect based at least in part on the velocity.15. The method of claim 10, further comprising determining a pressure ofthe user interaction and determining the haptic effect based at least inpart on the pressure.
 16. The method of claim 10, wherein the hapticoutput device is configured to displace the touch surface vertically,laterally, or vertically and laterally at an ultrasonic frequency. 17.The method of claim 16, wherein the haptic output device comprises oneor more of: a piezoelectric actuator, an electromagnetic actuator, anelectroactive polymer, a shape memory alloy, or a composite.
 18. Themethod of claim 10, wherein varying the coefficient of friction of thetouch surface comprises increasing the coefficient of friction of thetouch surface.
 19. A non-transitory computer-readable medium embodyingprogram code, which when executed by a processor is configured to causethe processor to: determine a position of a user interaction on a touchsurface; map the position of the user interaction to a location on agraphical user interface associated with the touch surface; determine ahaptic effect based in part on a color and a degree of shading at thelocation on the graphical user interface, wherein the haptic effect isconfigured to vary a coefficient of friction on the touch surface at theposition of the user interaction, and the degree of shading isassociated with the variance of the coefficient of friction; andtransmit a haptic signal to a haptic output device configured to outputthe haptic effect.
 20. The non-transitory computer-readable medium setforth in claim 19, wherein the haptic effect comprises a texture at ornear the position of the user interaction.